Are Citations to Academic Journal Articles a Measure of Quality or Something Else? An Exploratory Analysis with Emphasis on Design Science and IS Technical Research

One pervasive belief about scholarly citations that underlies several studies appearing in the IS literature is that the number of citations to a paper reflects the quality of the study. For example, a paper that receives 200 citations is perceived as having higher quality than another study that receives 50 citations. While most experts acknowledge that there are other factors that drive citations, such as the number of years a paper was available to be cited – or possibly “gaming” of citations by authors who cite themselves frequently (author self-citations) or over-zealous editors who incent prospective authors to cite their journals (journal self-citations), there is an underlying assumption that, barring such unscrupulous behavior, citations are a measure of research quality. This paper critically examines this assumption using a validated typology identifying 13 distinct subject areas that characterize IS research. Results show that papers on some topics (e.g., IT adoption and use; new measurement development and validation) consistently receive more citations than average – while some topics consistently receive below-average citation rates (e.g., IS development; IT project/risk management). Unless one assumes that all papers on a given topic are consistently of higher quality than all papers on other topics, our data suggest that a key driver of citations are the subject area of the study and, in turn, the size of the research community that conducts research on the topic.